Instrument navigation in endoscopic surgery during obscured vision

ABSTRACT

The application relates to the problem of navigating a surgical instrument (at  301, 311 ) towards a region-of-interest (at  312 ) in endoscopic surgery when an image ( 300 ) provided by the endoscope is obscured at least partly by obscuring matter (at  303 ), wherein the obscuring matter is a leaking body fluid, debris or smoke caused by ablation. To address this problem, a computer-implemented method is proposed, wherein, upon detecting that the image from the endoscope is at least partly obscured, a second image is determined based on a sequence of historic images and based on the current position and orientation of the endoscope. Furthermore, a virtual image ( 310 ) is generated based on the determined second image.

FIELD OF THE INVENTION

The invention relates to a computer-implemented method for animage-assisted medical application, a data processing system configuredto carry out the steps of this method, a computer program comprisinginstructions to cause the data processing system to execute the method,and a computer-readable medium having stored such computer program.

BACKGROUND OF THE INVENTION

Minimally invasive surgery is often guided by endoscopic vision. Theendoscope comprises a miniature camera, preferably a video camera, whichmay allow viewing the tissue in a body cavity that requires treatment.The vision provided by the endoscope's camera may facilitate navigatingan instrument towards a region-of-interest. During the treatment, thevision provided by the camera may be obscured. For example, the visionprovided by the camera may be obscured by a leaking body fluid such as ableeding caused by tissue resection, by debris, or by smoke fromablation.

For example in the case of a bleeding, the leaking blood vessel has tobe identified, which is difficult when the endoscopic vision is obscuredby the bleeding. The intervention may have to be interrupted untilendoscopic vision is cleared. Currently, water flushing near the distalend of the endoscope is applied in order to clear the vision. However,since the bleeding is not stopped, this may not be effective. Thus, thesurgical instrument may have to be maneuvered blindly, without knowingits location with respect to the tissue.

US 2015/313503 discloses a method for imaging with one or moreendoscopes. The method involves acquiring images with the endoscopes andcreating a virtual model from the images.

US 2013/018255 discloses a virtual endoscope image generation unit whichreceives a three dimensional medical image as an input and generates avirtual endoscope image representing a body cavity in real time.

US 2017/085762 discloses an endoscope system which includes an insertionportion, observation windows and an image processing portion. Oneobservation window is configured to acquire forward visual field imagesand another observation window is used to acquire lateral visual fieldimages.

US 2016/157726 discloses a method and program that generate a projectionimage from volume data representing a three-dimensional region includinga hollow organ.

SUMMARY OF THE INVENTION

It may be desirable to provide an improved method allowing to increasesafety of medical applications and/or treatments. For example, themethod may allow to safely navigate surgical instruments duringminimally invasive surgery while the image provided by the endoscope isat least partly obscured by a leaking body fluid such as a bleeding, bydebris, or by smoke caused by ablation.

This is achieved by the subject matter of the independent claims,wherein further embodiments are incorporated in the dependent claims andthe following description. It should be noted that any step, feature oraspect of the computer-implemented method, as described in thefollowing, equally applies to the data processing system configured tocarry out the steps of the method, the computer program, and thecomputer readable medium, as described in the following, and vice versa.

According to the present disclosure, a computer-implemented methodand/or medical method for an image-assisted medical application ispresented. The method may generally refer to a computer-implemented dataprocessing method. The method comprises the following steps: (i)acquiring a first image of at least one body part of a patient, thefirst image being captured with an endoscope and being associated withsensor data, the sensor data being indicative of a first position andorientation of the endoscope; (ii) detecting if the first image is atleast partly obscured by obscuring matter, such that the first imageincludes an image of a leaking body fluid, debris or smoke fromablation; (iii) determining, upon detecting that the first image is atleast partly obscured, a second image based on a sequence of historicimages and based on the first position and orientation of the endoscope,wherein the historic images of the sequence of historic images each haveimage capturing times earlier than the first image; and (iv) generatinga virtual image of the at least one body part based on the determinedsecond image.

The term “detecting if the first image is at least partly obscured byobscuring matter” may be understood as “detecting if the first imageincludes an image of obscuring matter, which obscuring matter obscuresthe at least one body part (or a portion thereof)”.

The first image may be received from the endoscope or from anintermediate device, which forwards the first image. Alternatively, thefirst image may be retrieved from a storage medium. The first imagevisualizes at least one body part of a patient, wherein the patient maybe a human or an animal.

The endoscope may be a laparoscope, cystoscope, nephroscope,bronchoscope, arthroscope, or colonoscope. Preferably, the endoscope isa rigid endoscope so that the position and orientation of the camera atthe distal end of the endoscope can be inferred from the position andorientation of a visible proximal end of the endoscope. The firstposition and orientation of the endoscope may be defined as the positionand orientation of the endoscopic camera.

The sensor data may have been recorded by a tracking system such as acamera-based tracking system. The camera-based tracking system maycomprise one or more cameras, which capture images from one or morepositions and orientations. At least a proximal end of the endoscope maybe visible in one or more images captured by the camera-based trackingsystem. The first position and orientation of the endoscope may bedetermined based on this visualization of the proximal end of theendoscope in one or more camera images, based on information about theshape of the endoscope, based on information about the shape of amarker-plate, which may be attached to the endoscope, and/or based oninformation about the geometry of the cameras of the tracking system.The position and orientation of the endoscope may be defined withrespect to a coordinate system that is stationary relative to theposition and orientation of the patient or relative to the position andorientation of an operating table, on which the patient may be lying.

The sensor data may be received from the tracking system or from anintermediate device, which forwards the sensor data. Alternatively, thesensor data may be retrieved from a storage medium. The sensor data maybe the raw recorded sensor data, or it may be a processed versionthereof. Preferably, the sensor data is recorded by the tracking systemat the same time when the endoscope captures the first image. Hence, thefirst position and orientation of the endoscope determined based on thesensor data may be an estimate of the image capturing position andorientation of the first image. However, the endoscope may not becoupled to the tracking system, so the capturing time of the first imagemay diverge from the time of recording the sensor data. The differencein time may be negligible as long as the position and orientation of theendoscope does not change significantly in the meantime.

It is detected if the first image is at least partly obscured byobscuring matter, wherein the obscuring matter is a leaking body fluid,debris or smoke caused by ablation. The term ‘obscuring matter’ is usedhere and in the following for a leaking body fluid, debris or smokecaused by ablation. The leaking body fluid may be but not limited to ableeding, bile or gall leaking out of a bile duct or an exudate (pus) ofa site of inflammation. Furthermore, for the sake of brevity, an imageis referred to herein as being ‘obscured’ when at least a part of theimage is obscured by a leaking body fluid, debris or smoke fromablation. The obscuring matter may obscure a part of the first image, sothat the first image may not show for example an instrument and/or aregion-of-interest. Thus, due to the obscuring matter, it may bedifficult or impossible to securely navigate the instrument towards theregion-of-interest based on the first image.

When it is detected that the first image is at least partly obscured,the first position and orientation of the endoscope may be determinedbased on the sensor data, and a second image may be determined based onthe sequence of historic images and based on the first position andorientation of the endoscope. The sequence of historic images maycomprise endoscopic images, which have been captured by the endoscopebefore capturing the first image.

Determining the second image may comprise, for example, retrieving thesecond image from the sequence of historic images such that a measurefor the difference between the first position and orientation of theendoscope and the image capturing position and orientation of the secondimage is small. Thus, the second image may be identical with one of thehistoric images from the sequence of historic images. The imagecapturing position and orientation of the second image may differ fromthe first position and orientation of the endoscope.

Alternatively, determining the second image may comprise, for example,generating a three-dimensional model of the patient's anatomy from thesequence of two-dimensional historic images. The second image may thenbe determined based on the three-dimensional model and the firstposition and orientation of the endoscope. In this case, the secondimage may differ from all historic images in the sequence of historicimages, and its associated image capturing position and orientation maybe identical with the first position and orientation of the endoscope.

A virtual image is then generated based on the second image. The virtualimage may be equal to the second image. However, the generation of thevirtual image may also comprise various transformations of the secondimage. For example, the second image may be transformed to adjust itsassociated image capturing position and orientation. Furthermore, thesecond image may be augmented with a rendering of an instrument. Inaddition, the second image may be augmented with image data from anX-ray imaging system, a magnetic resonance imaging (MRI) system, asingle-photon emission computerized tomography (SPECT) system or anultrasound device.

The virtual image may not be based on image data of the first image.Hence, even when the first image is obscured by obscuring matter, thevirtual image may provide a virtual endoscopic view, which is notobscured by obscuring matter. The method may comprise displaying thevirtual image to provide an unobscured view to a user.

In an example, the sensor data is also indicative of a position andorientation of an instrument, and the method further comprises thefollowing steps: Determining a second position and orientation of theinstrument based on the sensor data, and augmenting the virtual imagewith a rendering of the instrument in accordance with the secondposition and orientation of the instrument and in accordance with animage capturing position and orientation of the virtual image.

The instrument may be any instrument used for minimally invasivesurgery, including, but not limited to, forceps, spatulas, retractors,dilators, graspers, sutures, visualizing scopes, cutter instruments suchas trocars and rasps, electrosurgical instruments, guiding devices,surgical staplers, needles, catheters and inflation systems.

The sensor data may have been recorded by a tracking system such as acamera-based tracking system. The camera-based tracking system maycomprise a plurality of cameras, which capture images from a pluralityof positions and orientations. At least a proximal end of the instrumentmay be visible in one or more images captured by the camera-basedtracking system. The second position and orientation of the instrumentmay be determined based on this visualization of the proximal end of theinstrument in one or more camera images, based on information about theshape of the instrument, based on information about the shape of amarker-plate, which may be attached to the instrument, and/or based oninformation about the geometry of the cameras of the tracking system.

The virtual image may be augmented with a rendering of the instrument inaccordance with the second position and orientation of the instrumentand in accordance with an image capturing position and orientation ofthe virtual image, so that the virtual image provides a virtualendoscopic view, which shows the instrument in accordance with itscurrent position and orientation. The instrument may be rendered basedon a three-dimensional model of the instrument.

In another example, determining the second image comprises retrievingthe second image from the sequence of historic images based on a measurefor a difference between the first position and orientation of theendoscope and an image capturing position and orientation of the secondimage.

The second image may be retrieved from the sequence of historic imagessuch that the measure for the difference between the first position andorientation of the endoscope and the image capturing position andorientation of the second image is small or such that the measure forthe difference is minimized over the image capturing positions andorientations of the historic images in the sequence of historic images.The measure for the difference between the first position andorientation of the endoscope and the image capturing position andorientation of the second image may be a function, e.g. a weighted sum,of a measure for the difference between the first position of theendoscope and the image capturing position of the second image and ameasure for the difference between the first orientation of theendoscope and the image capturing orientation of the second image.

Additionally or alternatively, the second image may be retrieved fromthe sequence of historic images based on the difference between theimage capturing times of the first and second images. For example, thesecond image may be retrieved from the sequence of historic images suchthat a measure for the difference between the first position andorientation of the endoscope and the image capturing position andorientation of the second image is small subject to the constraint thatthe difference between the image capturing times of the first and secondimages does not exceed a threshold.

In another example, the determination of the second image is also basedon a determination for one or more images if they are at least partlyobscured by obscuring matter to ensure that the determined second imageis not obscured by obscuring matter.

For example, determining the second image may comprise retrieving thesecond image from the sequence of historic images based on a measure fora difference between the first position and orientation of the endoscopeand an image capturing position and orientation of the second image andbased on a determination for one or more images if they are obscured byobscuring matter to ensure that the retrieved second image is notobscured by such matter. The sequence of historic images may compriseobscured and non-obscured historic images, and it may be determined forone or more historic images from the sequence of historic images if theyare obscured by obscuring matter to ensure that the retrieved secondimage is not obscured by such matter. Hence, the obscured historicimages may be excluded when retrieving the second image. Alternatively,an image may be included in the sequence of historic images only if itis determined that this image is not obscured by obscuring matter, sothe sequence of historic images may comprise only non-obscured historicimages.

Alternatively, a three-dimensional model of the patient's anatomy may begenerated from the sequence of two-dimensional historic images, and thesecond image may be determined based on the three-dimensional model andthe first position and orientation of the endoscope. The sequence ofhistoric images may comprise obscured and non-obscured historic images,and it may be determined for one or more historic images from thesequence of historic images if they are obscured by obscuring matter toensure that parts of historic images, which are obscured by such matter,are not taken into account for generating the three-dimensional model,so that also the second image is not obscured by obscuring matter.Alternatively, before including an image in the sequence of historicimages, it may be determined if the image is obscured by obscuringmatter to ensure that the sequence of historic images comprises onlynon-obscured historic images.

In another example, generating the virtual image comprises transformingthe second image by means of the structure from motion technique,wherein the virtual image corresponds to an image capturing position andorientation equal to the first position and orientation of theendoscope.

The image capturing position and orientation of the second image maydiverge from the first position and orientation of the endoscope. Thefirst position and orientation of the endoscope may be equal or close tothe current position and orientation of the endoscope. Hence, bytransforming the second image such that the virtual image has an imagecapturing position and orientation equal to the first position andorientation of the endoscope, a virtual image may be generated, whichcorresponds to the current position and orientation of the endoscope.The second image may be transformed by means of the structure frommotion technique or by means of any other image processing techniquesuitable for adjusting the image capturing position and orientation ofthe second image.

In another example, the method further comprises displaying an indicatorfor a difference between the first position and orientation of theendoscope and an image capturing position and orientation of the secondimage.

When the virtual image has been generated by transforming the secondimage such that the virtual image has an image capturing position andorientation equal to the first position and orientation of theendoscope, for instance by means of the structure from motion technique,then displaying the indicator for the difference between the firstposition and orientation of the endoscope and the image capturingposition and orientation of the second image may provide an indicationfor the reliability of the virtual image. Furthermore, displaying theindicator for the difference between the first position and orientationof the endoscope and the image capturing position and orientation of thesecond image may indicate to a user how to adjust the current positionand orientation of the endoscope to reduce the difference between thecurrent position and orientation of the endoscope and the imagecapturing position and orientation of the second image, therebyincreasing the reliability of the virtual image.

Alternatively, when the virtual image has an image capturing positionand orientation equal to that of the second image, then the indicatorfor the difference between the first position and orientation of theendoscope and the image capturing position and orientation of the secondimage may indicate to the user how to adjust the current position andorientation of the endoscope such that the current position andorientation of the endoscope becomes equal to the image capturingposition and orientation of the virtual image.

In another example, the method further comprises displaying the firstimage from the endoscope and the virtual image next to each other.

The first image may provide an endoscopic view, which may be obscured byobscuring matter, whereas the virtual image may provide a virtualunobscured endoscopic view. Displaying the first and virtual images nextto each other may allow the user to compare these images and to assessthe reliability of the virtual image. The first and virtual images maybe displayed separately, for example next to each other, on top of eachother, or otherwise shifted relative to each other. The first andvirtual images may be displayed on the same screen or on differentscreens.

In another example, the historic images of the sequence of historicimages are endoscopic images.

For example, when it is detected that the first image is not obscured byobscuring matter, this image may be included in the sequence of historicimages after detecting that it is not obscured. On the other hand, whenit is detected that the first image is at least partly obscured byobscuring matter, this image may be included in the sequence of historicimages after generating the virtual image.

The sequence of historic images may comprise images, which are obscuredby obscuring matter, as well as images, which are not obscured by suchmatter. The sequence of historic images may be stored in a database.Hence, the method may comprise storing the first image in the database.Furthermore, the method may comprise storing the first position andorientation of the endoscope in association with the first image.Furthermore, the method may comprise storing the image capturing time ofthe first image in association with the first image. Furthermore, themethod may comprise storing an indicator in association with the firstimage, wherein the indicator indicates if the first image is obscured byobscuring matter. Furthermore, the method may comprise storing anotherindicator in association with the first image, wherein the anotherindicator indicates if the first image comprises a visualization of aninstrument.

Alternatively, the sequence of historic images may comprise onlyunobscured images. Thus, the method may comprise storing the first imagein the database upon detecting that the first image is not obscured byobscuring matter.

In another example, detecting if the first image is obscured byobscuring matter comprises detecting if the first image is obscured by ableeding, wherein detecting if the first image is obscured by a bleedingcomprises determining a size of a red section in the first image and/ordetermining a contrast in at least a part of the first image.

Detecting if the first image is obscured by a bleeding may comprisedetecting if the size of a red section increased over time and/or bydetecting if the image contrast in at least a part of the imagedecreased over time. This may be accomplished by comparing the firstimage with an earlier image having an image capturing time earlier thanthe first image. For example, the earlier image may be a historic imagefrom the sequence of historic images.

Hence, detecting if the first image is obscured by a bleeding maycomprise detecting if the size of a red section in the earlier image issmaller than the size of a red section in the first image. Additionallyor alternatively, detecting if the first image is obscured by a bleedingmay comprise detecting if the image contrast in at least a part of theearlier image is higher than the image contrast in at least a part ofthe first image.

More specifically, detecting if the size of a red section in the earlierimage is smaller than the size of a red section in the first image maycomprise determining a red section in the earlier image, determining ameasure for the size of the red section in the earlier image,determining a red section in the first image, determining a measure forthe size of the red section in the first image, and determining if thedifference between the measures for the sizes of the red sections in thefirst and earlier images exceeds a threshold.

Further, detecting if the image contrast in at least a part of theearlier image is higher than the image contrast in at least a part ofthe first image may comprise determining a part of the earlier image,determining a measure for the image contrast in the part of the earlierimage, determining a part of the first image, determining a measure forthe image contrast in the part of the first image, and determining ifthe difference between the measures for the image contrast in theearlier and first images exceeds a threshold.

The detection if the first image is obscured by a bleeding may beperformed by means of a neural network, which may be trained usingstandard machine learning techniques. One input parameter of the neuralnetwork may be based on the measure for the size of the red section inthe earlier image and/or based on the measure for the size of the redsection in the first image. Another input parameter of the neuralnetwork may be based on the measure for the image contrast in the partof the earlier image and/or based on the measure for the image contrastin the part of the first image. Another input parameter of the neuralnetwork may be based on the image capturing time of the earlier imageand/or based on the image capturing time of the first image. Anotherinput parameter of the neural network may be based on the imagecapturing position and orientation of the earlier image and/or based onthe image capturing position and orientation of the first image. Forexample, one input parameter of the neural network may be based on theprojection of the vector from the image capturing position of theearlier image to the image capturing position of the first image ontothe direction of the image capturing orientation of the earlier image.

More generally, detecting if the first image is obscured by a leakingbody fluid or by smoke caused by ablation may comprise detecting if theimage contrast in at least a part of the image decreased over time.Again, this may be accomplished by comparing the first image with anearlier image such as a historic image from the sequence of historicimages. Hence, detecting if the first image is obscured by a leakingbody fluid or by smoke from ablation may comprise detecting if the imagecontrast in at least a part of the earlier image is higher than theimage contrast in at least a part of the first image.

The detection if the first image is obscured by a leaking body fluid orby smoke from ablation may again be performed by means of a neuralnetwork. One input parameter of the neural network may be based on ameasure for the image contrast in a part of the earlier image and/orbased on a measure for the image contrast in a part of the first image.Another input parameter of the neural network may be based on the imagecapturing time of the earlier image and/or based on the image capturingtime of the first image. Another input parameter of the neural networkmay be based on the image capturing position and orientation of theearlier image and/or based on the image capturing position andorientation of the first image. For example, one input parameter of theneural network may be based on the projection of the vector from theimage capturing position of the earlier image to the image capturingposition of the first image onto the direction of the image capturingorientation of the earlier image.

In another example, the method further comprises determining a positionof an origin of obscuring matter based on the sequence of historicimages, and/or indicating a position of an origin of obscuring matter inthe virtual image.

Determining a position of an origin of obscuring matter may comprisesearching for a first historic image in the sequence of historic images,wherein the first historic image is not obscured by obscuring matter,whereas the next-in-time historic image in the sequence of historicimages is obscured the obscuring matter. The position of the obscuringmatter in the next-in-time historic image may then provide an indicationfor the position of the origin of the obscuring matter.

When the obscuring matter is a leaking body fluid such as a bleeding,then indicating the position of the origin of the leaking body fluid inthe virtual image may allow a user to navigate an instrument towards theorigin of the leaking body fluid in order to stop the leakage of thebody fluid. Similarly, when the obscuring matter is debris or smoke fromablation, then indicating the position of the origin of the debris orthe smoke from ablation in the virtual image may inform a user wherecertain treatments may be required.

In another example, the sensor data is received from a camera-basedtracking system, and/or the first position and orientation of theendoscope is determined based on a marker plate attached to theendoscope.

The camera-based tracking system may comprise one or more cameras, whichmay capture images from one or more positions and orientations. At leasta proximal end of the endoscope may be visible in one or more imagescaptured by the camera-based tracking system. The first position andorientation of the endoscope may be determined based on thisvisualization of the proximal end of the endoscope in one or more cameraimages and based on information about the geometry of the trackingcameras. Additionally, information about the shape of the endoscope maybe utilized for determining the position and orientation of theendoscope.

When a marker plate is attached to the endoscope, the position andorientation of the endoscope may be determined based on a single imagefrom a single camera. However, a camera-based tracking system comprisinga plurality of cameras still provides benefits as compared to a singlecamera system. For example, in the case of a plurality of cameras, it ismore likely that the endoscope is visible in at least one of the cameraimages as compared to a single camera system, so the determination ofthe first position and orientation of the endoscope is more reliable.Additionally or alternatively, the plurality of cameras and theassociated image processing algorithms may be configured to provide astatistically more accurate determination of the position andorientation of the endoscope as compared to a single camera system.

In another example, the method further comprises acquiring a thirdimage, the third image being captured with an X-ray imaging system, acomputed tomography (CT) system, a single-photon emission computerizedtomography (SPECT) system, a magnetic resonance imaging (MM) system oran ultrasound device, and adding data from the third image in thevirtual image.

For example, the third image may be an X-ray image, which may beacquired by receiving the image from an X-ray imaging system, byreceiving the image from an intermediate device, which forwards theimage, or by retrieving the image from a storage medium. The X-ray imagemay be a two-dimensional image or a three-dimensional model determinedby a computed tomography (CT) system. The third image may be associatedwith a coordinate system of the X-ray imaging system, and the method maycomprise transforming the third image based on the coordinate system ofthe X-ray imaging system and based on the image capturing position andorientation of the virtual image. Alternatively, the transformationbased on the coordinate system of the X-ray imaging system and based onthe image capturing position and orientation of the virtual image mayhave been performed, for example by the X-ray imaging system, before thethird image is acquired. Hence, the method may comprise adding data fromthe third image or the transformed third image in the virtual image inaccordance with the image capturing position and orientation of thevirtual image.

Similarly, the third image may be a SPECT image, which may be acquiredby receiving the image from a SPECT system, by receiving the image froman intermediate device, which forwards the image, or by retrieving theimage from a storage medium. The SPECT image may be a two-dimensionalimage or a three-dimensional model determined by a tomographicreconstruction. The third image may be associated with a coordinatesystem of the SPECT system, and the method may comprise transforming thethird image based on the coordinate system of the SPECT system and basedon the image capturing position and orientation of the virtual image.Alternatively, the transformation based on the coordinate system of theSPECT system and based on the image capturing position and orientationof the virtual image may have been performed, for example by the SPECTsystem, before the third image is acquired. Hence, the method maycomprise adding data from the third image or the transformed third imagein the virtual image in accordance with the image capturing position andorientation of the virtual image.

Similarly, the third image may be an MRI image, which may be acquired byreceiving the image from an Mill system, by receiving the image from anintermediate device, which forwards the image, or by retrieving theimage from a storage medium. The Mill image may be a two- orthree-dimensional image. The third image may be associated with acoordinate system of the Mill system, and the method may comprisetransforming the third image based on the coordinate system of the MMsystem and based on the image capturing position and orientation of thevirtual image. Alternatively, the transformation based on the coordinatesystem of the MM system and based on the image capturing position andorientation of the virtual image may have been performed, for example bythe MM system, before the third image is acquired. Hence, the method maycomprise adding data from the third image or the transformed third imagein the virtual image in accordance with the image capturing position andorientation of the virtual image.

Furthermore, the third image may be an ultrasound image, which may beacquired by receiving the image from an ultrasound device, by receivingthe image from an intermediate device, which forwards the image, or byretrieving the image from a storage medium. The ultrasound image may bea two- or three-dimensional image. The method may comprise determining athird position and orientation of the ultrasound device based on thesensor data. Furthermore, the method may comprise transforming the thirdimage based on the third position and orientation of the ultrasounddevice and based on the image capturing position and orientation of thevirtual image. Alternatively, the transformation based on third positionand orientation of the ultrasound device and based on the imagecapturing position and orientation of the virtual image may have beenperformed before the third image is acquired, for example by theultrasound device. Hence, the method may comprise adding data from thethird image or the transformed third image in the virtual image inaccordance with the image capturing position and orientation of thevirtual image.

In another example, the determination of the second image is also basedon a determination for one or more images if they depict an instrumentto ensure that the determined second image does not depict aninstrument.

Herein, an image is said to depict an instrument when the image depictsat least a part of the instrument.

Determining the second image may comprise, for example, retrieving aparticular image from the sequence of historic images based on a measurefor a difference between the first position and orientation of theendoscope and an image capturing position and orientation of theparticular image. The second image may be determined based on theretrieved particular image and based on a determination if theparticular image depicts an instrument to ensure that the second imagedoes not show an instrument. For example, when the particular image doesnot depict an instrument, the second image may be equal to theparticular image. On the other hand, when the particular image depictsan instrument, the position and orientation of the instrument at theimage capturing time of the particular image may be different from thesecond position and orientation of the instrument determined based onthe sensor data. Then, the particular image may be transformed based onthe sequence of historic images to remove the visualization of theinstrument from the particular image.

Alternatively, a three-dimensional model may be generated from thesequence of two-dimensional historic images, and the second image may bedetermined based on the three-dimensional model and the first positionand orientation of the endoscope. The sequence of historic images maycomprise images, which depict an instrument, and other images, which donot depict an instrument. It may be determined for one or more historicimages from the sequence of historic images if they depict an instrumentto ensure that parts of historic images, which depict an instrument, arenot taken into account for generating the three-dimensional model, sothat also the determined second image does not depict an instrument.Alternatively, before including an image in the sequence of historicimages, it may be determined if the image shows an instrument to ensurethat the sequence of historic images comprises only historic images,which do not show an instrument.

It is emphasized, however, that the invention as described above and inthe following does not involve or in particular comprise or encompass aninvasive step which would represent a substantial physical interferencewith the body of a patient requiring professional medical expertise tobe carried out and entailing a substantial health risk even when carriedout with the required professional care and expertise. In particular,the invention does not involve or in particular comprise or encompassany surgical or therapeutic activity. The invention is instead directedas applicable to any non-invasive medical application and merely relatesto a data processing method. For this reason alone, no surgical ortherapeutic activity and in particular no surgical or therapeutic stepis necessitated or implied by carrying out the invention.

According to the present disclosure, also a data processing system ispresented. The data processing system is configured to carry out thesteps of any of the methods according to the present invention.

The data processing system may be connected to an endoscope.Furthermore, the data processing system may be connected to a trackingsystem, for example, a camera-based tracking system. The data processingsystem may further comprise a storage medium for storing the sequence ofhistoric images in a database. The database may further comprise, for atleast some of the historic images, image capturing positions andorientations, indicators of image capturing times, indicators indicatingif an image is at least partly obscured, and/or indicators indicating ifan image depicts an instrument. Furthermore, the data processing systemmay be connected to or may comprise one or more computer screens.

According to the present disclosure, also a computer program ispresented, wherein the computer program comprises instructions to causethe data processing system as defined in the independent claims toexecute any one of the methods according to the present invention whenthe computer program is run on the data processing system.

According to the present disclosure, also a computer-readable medium ispresented, wherein the computer-readable medium stores the computerprogram as defined in the independent claims.

It shall be understood that the computer-implemented method for animage-assisted medical application, the data processing systemconfigured to carry out the steps of the method, the computer programfor causing the data processing system to execute the method, and thecomputer readable medium having stored such computer program havesimilar and/or identical preferred embodiments, in particular, asdefined in the dependent claims. It shall be understood further that apreferred embodiment of the invention can also be any combination of thedependent claims with the respective independent claim.

These and other aspects of the present invention will become apparentfrom and be elucidated with reference to the embodiments describedhereinafter.

BRIEF DESCRIPTION OF THE DRAWINGS

Exemplary embodiments of the invention will be described in thefollowing with reference to the accompanying drawings:

FIG. 1 shows basic steps of an example of a computer-implemented methodfor an image-assisted medical application.

FIG. 2 shows schematically and exemplarily a non-obscured endoscopicimage (FIG. 2a ) and a virtual computer-generated image (FIG. 2b ).

FIG. 3 shows schematically and exemplarily an obscured endoscopic image(FIG. 3a ) and a virtual image (FIG. 3b ) generated by thecomputer-implemented method as illustrated by FIG. 1.

FIG. 4 shows schematically and exemplarily an embodiment of a dataprocessing system for carrying out the computer-implemented method foran image-assisted medical application.

DETAILED DESCRIPTION OF EMBODIMENTS

FIG. 1 shows basic steps of an example of a computer-implemented methodfor an image-assisted medical application. The method comprises thefollowing steps:

In a first step S1, a first image of at least one body part of a patientis acquired, the first image being captured with an endoscope and beingassociated with sensor data, the sensor data being indicative of a firstposition and orientation of the endoscope. The first image may bereceived from the endoscope or from an intermediate device, whichforwards the first image. Alternatively, the first image may beretrieved from a storage medium.

The sensor data may have been recorded by a tracking system such as acamera-based tracking system. The sensor data may be received from thetracking system or from an intermediate device, which forwards thesensor data. Alternatively, the sensor data may be retrieved from astorage medium. The camera-based tracking system may comprise one ormore cameras, which capture images from one or more positions andorientations. Hence, the sensor data may comprise one or more cameraimages. At least a proximal end of the endoscope may be visible in oneor more of these images. The first position and orientation of theendoscope may be determined based on this visualization of the proximalend of the endoscope in one or more camera images. The endoscope ispreferably a rigid endoscope so that the position and orientation of thecamera at the distal end of the endoscope can be inferred from theposition and orientation of the visible proximal end of the endoscope.The first position and orientation of the endoscope may be defined asthe position and orientation of the endoscopic camera. Preferably, thesensor data is recorded by the tracking system at approximately the sametime when the endoscope captures the first image. Hence, the firstposition and orientation of the endoscope may be an estimate of theimage capturing position and orientation of the first image.

In a second step S2, it is detected if the first image is at leastpartly obscured by obscuring matter, wherein the obscuring matter is aleaking body fluid, debris or smoke from ablation. The obscuring mattermay obscure a part of the first image, so that the first image may notshow for example an instrument and/or a region-of-interest. Thus, due tothe obscuring matter, it may be difficult or impossible to securelynavigate the instrument towards the region-of-interest based on thefirst image.

In a third step S3, upon detecting that the first image is at leastpartly obscured, a second image is determined based on a sequence ofhistoric images and based on the first position and orientation of theendoscope, wherein the historic images of the sequence of historicimages each have image capturing times earlier than the first image.When it is detected that the first image is at least partly obscured,the first position and orientation of the endoscope may be determinedbased on the sensor data, and the second image may be determined basedon the sequence of historic images and based on the first position andorientation of the endoscope. The sequence of historic images maycomprise endoscopic images, which have been captured by the endoscopebefore capturing the first image.

In a fourth step S4, a virtual image of the at least one body part isgenerated based on the determined second image. The virtual image may beequal to the second image. However, the generation of the virtual imagemay also comprise various transformations of the second image. Forexample, the second image may be transformed to adjust its associatedimage capturing position and orientation. Furthermore, the second imagemay be augmented with a rendering of an instrument. In addition, thesecond image may be augmented with image data from an X-ray imagingsystem, a magnetic resonance imaging (Mill) system, a single-photonemission computerized tomography (SPECT) system or an ultrasound device.

Even when the first image is obscured by obscuring matter, the virtualimage may provide a virtual endoscopic view, which is not obscured bysuch matter.

Note that step S3 is performed upon detecting that the first image isobscured by obscuring matter. Additionally or alternatively, thedetermination of the second image and the generation of the virtualimage may be performed upon receiving a request from a user to generatethe virtual image. Furthermore, in another method, the second image maybe determined and the virtual image may be generated even if the firstimage is not obscured by obscuring matter and even if the generation ofthe virtual image is not requested. When the first image is notobscured, the virtual image may still be generated to assess thereliability/accuracy of the generated virtual image.

FIG. 2 shows schematically and exemplarily a non-obscured endoscopicimage 200 and a virtual image 210. The non-obscured endoscopic image 200comprises a presentation of an instrument 201 and a presentation of alesion 202, wherein the lesion may require treatment. The non-obscuredimage 200 may be associated with sensor data, which may allowdetermining the first position and orientation of the endoscope as anestimate for the image capturing position and orientation of the image200.

The virtual image 210 may have been generated based on a second image,wherein the second image may have been determined based on a sequence ofhistoric images and based on the first position and orientation of theendoscope. The historic images of the sequence of historic images mayeach have image capturing times earlier than the image 200. The virtualimage 210 shows a visualization of the instrument 211 and avisualization of the lesion 212. Since the image 200 is not obscured byobscuring matter, the virtual image 210 may be essentially identicalwith the image 200.

FIG. 3 shows schematically and exemplarily an obscured endoscopic image300 and a virtual image 310. The scenario of FIG. 3 differs from thescenario of FIG. 2 in that the image 300 is partly obscured by obscuringmatter 303, which may be bleeding. The image 300 further comprises apresentation of an instrument 301. The obscured image 300 may beassociated with sensor data, which may allow determining the firstposition and orientation of the endoscope as an estimate of the imagecapturing position and orientation of the image 300.

FIG. 4 shows schematically and exemplarily an embodiment of a dataprocessing system 400 for carrying out the computer-implemented method100 for an image-assisted medical application. The data processingsystem may comprise a processor 401 and a storage medium 402. The dataprocessing system may be connected to an endoscope 404 and may beconfigured for acquiring a first image of at least one body part of apatient, the first image being captured with the endoscope. Furthermore,the data processing system may be connected to a tracking system 405,for example a camera-based tracking system. The data processing systemmay be configured to acquire sensor data, the sensor data being recordedby the tracking system. The data processing system may be configured todetermine the first position and orientation of the endoscope based onthe sensor data. The first position and orientation of the endoscope maybe an estimate of the image capturing position and orientation of thefirst image. Furthermore, the data processing system may be configuredto detect if the first image is at least partly obscured by obscuringmatter, wherein the obscuring matter is a leaking body fluid, debris orsmoke from ablation. In addition, the data processing system may beconfigured to determine, upon detecting that the first image is at leastpartly obscured, a second image based on the first position andorientation of the endoscope and based on a sequence of historic images403, which may be stored on the storage medium of the data processingsystem. Alternatively, the sequence of historic images may be stored onan external storage medium or on a server connected to the dataprocessing system. Moreover, the data processing system may beconfigured for generating a virtual image of the at least one body partbased on the determined second image.

The data processing system may be connected to an X-ray imaging system407, and the data processing system may be configured to acquire anX-ray image being captured with the X-ray imaging system and to add datafrom the X-ray image in the virtual image. Additionally oralternatively, the data processing system may be connected to a CTsystem, a SPECT system, an Mill system or an ultrasound device, and thedata processing system may be configured to add image data from such asystem or device in the virtual image (not shown in the figure).

The data processing system may also be connected to a display 406, andthe data processing system may be configured to display the first andvirtual images next to each other by means of the display.

It has to be noted that embodiments of the invention are described withreference to different subject matters. However, a person skilled in theart will gather from the above and the following description that,unless otherwise notified, in addition to any combination of featuresbelonging to one type of subject matter also any combination betweenfeatures relating to different subject matters is considered to bedisclosed with this application. However, all features can be combinedproviding synergetic effects that are more than the simple summation ofthe features.

While the invention has been illustrated and described in detail in thedrawings and foregoing description, such illustration and descriptionare to be considered illustrative or exemplary and not restrictive. Theinvention is not limited to the disclosed embodiments. Other variationsto the disclosed embodiments can be understood and effected by thoseskilled in the art in practicing a claimed invention, from a study ofthe drawings, the disclosure, and the dependent claims.

In the claims, the word “comprising” does not exclude other elements orsteps, and the indefinite article “a” or “an” does not exclude aplurality. A single processor or other unit may fulfil the functions ofseveral items re-cited in the claims. The mere fact that certainmeasures are re-cited in mutually different dependent claims does notindicate that a combination of these measures cannot be used toadvantage. Any reference signs in the claims should not be construed aslimiting the scope.

1. A computer-implemented method for an image-assisted medicalapplication, the method comprising: acquiring (S1) a first image of atleast one body part of a patient, the first image being captured with anendoscope and being associated with sensor data, the sensor data beingindicative of a first position and orientation of the endoscope;detecting (S2) if the first image includes an image of obscuring matter,such that the first image includes an image of a leaking body fluid,debris or smoke from ablation, wherein the image of the obscuring matterobscures the at least one body part, or a portion thereof; determining(S3), upon detecting that the first image includes an image of obscuringmatter, a second image based on a sequence of historic images and basedon the first position and orientation of the endoscope, wherein thehistoric images of the sequence of historic images each have imagecapturing times earlier than the first image; and generating (S4) avirtual image of the at least one body part based on the determinedsecond image.
 2. The method of claim 1, wherein the sensor data is alsoindicative of a position and orientation of an instrument, and whereinthe method further comprises: determining a second position andorientation of the instrument based on the sensor data; and augmentingthe virtual image with a rendering of the instrument in accordance withthe second position and orientation of the instrument and in accordancewith an image capturing position and orientation of the virtual image.3. The method of claim 1, wherein determining (S3) the second imagecomprises retrieving the second image from the sequence of historicimages based on a measure for a difference between the first positionand orientation of the endoscope and an image capturing position andorientation of the second image.
 4. The method of claim 1, whereindetermining (S3) the second image is based on a determination for one ormore images if they include an image of the obscuring matter to ensurethat the second image does not include an image of the obscuring matter.5. The method of claim 1, wherein generating (S4) the virtual imagecomprises transforming the second image by means of the structure frommotion technique, wherein the virtual image corresponds to an imagecapturing position and orientation equal to the first position andorientation of the endoscope.
 6. The method of claim 1, furthercomprising: displaying an indicator for a difference between the firstposition and orientation of the endoscope and an image capturingposition and orientation of the second image.
 7. The method of claim 1,further comprising: displaying the first image from the endoscope andthe virtual image next to each other.
 8. The method of claim 1, whereinthe historic images of the sequence of historic images are endoscopicimages.
 9. The method of claim 1, wherein detecting (S2) if the firstimage includes an image of by the obscuring matter comprises detectingif the first image includes an image of a bleeding, wherein detecting ifthe first image includes an image of a bleeding comprises determining asize of a red section in the first image and/or determining a contrastin at least a part of the first image.
 10. The method of claim 1,further comprising: determining a position of an origin of the obscuringmatter based on the sequence of historic images; and/or indicating aposition of an origin of the obscuring matter in the virtual image. 11.The method of claim 1, wherein the sensor data is received from acamera-based tracking system; and/or wherein the first position andorientation of the endoscope is determined based on a marker plateattached to the endoscope.
 12. The method of claim 1, furthercomprising: acquiring a third image, the third image being captured withan X-ray imaging system, a computed tomography system, a single-photonemission computerized tomography system, a magnetic resonance imagingsystem or an ultrasound device; and adding data from the third image inthe virtual image.
 13. A data processing system configured to carry outthe steps of claim
 1. 14. A computer program comprising instructions tocause the data processing system to execute the steps of claim
 1. 15. Acomputer-readable medium having stored thereon the computer program ofclaim 14.